alert: Send alert for ha'ed vm's by ravening · Pull Request #5664 · apache/cloudstack

ravening · 2021-11-04T11:42:51Z

Description

When ha is performed on vm's send the alert for it so that
its for admins to know which vm's got ha'ed else its time
consuming to get those details from logs

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

DaanHoogland

clgtm, but would the subject exactly the same as the content be helpful? can we add more info to the mail content?

ravening · 2021-11-04T12:32:17Z

clgtm, but would the subject exactly the same as the content be helpful? can we add more info to the mail content?

@DaanHoogland if you have any content then let me know.. i can add that

DaanHoogland · 2021-11-04T14:15:08Z

clgtm, but would the subject exactly the same as the content be helpful? can we add more info to the mail content?

@DaanHoogland if you have any content then let me know.. i can add that

source and target host? time down? time up? down time duration?
just suggestions @ravening . What would you need to know as operator. As said, clgtm

yadvr · 2021-11-18T08:22:36Z

@blueorangutan package

blueorangutan · 2021-11-18T08:23:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-11-18T10:28:42Z

Packaging result: ✖️ el7 ✔️ el8 ✖️ debian ✖️ suse15. SL-JID 1723

yadvr · 2021-11-23T11:20:21Z

@blueorangutan package

blueorangutan · 2021-11-23T11:21:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-11-23T13:25:58Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✔️ suse15. SL-JID 1771

yadvr · 2021-12-30T10:50:54Z

@blueorangutan package

blueorangutan · 2021-12-30T10:53:08Z

@rohityadavcloud a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-12-30T12:42:21Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 2039

GabrielBrascher

Thanks for the PR @ravening.
I've proposed a few changes that, in my opinion, can make this PR even more helpful :-)

GabrielBrascher · 2021-12-30T16:05:19Z


            VMInstanceVO started = _instanceDao.findById(vm.getId());
            if (started != null && started.getState() == VirtualMachine.State.Running) {
+                String message = String.format("HA on VM: %s", started.getHostName());


I like the idea of enhancing the alerts.
What do you think of changing from Ha on VM: VM Name to Ha Starting VM: VM Name (i-2-13477-VM).

Thus, making it String.format("HA on VM: %s (%s)", started.getHostName(), started.getInstanceName());.

@ravening Do you want to add instance name to the alert message (as advised)?

GabrielBrascher · 2021-12-30T16:13:51Z

            VMInstanceVO started = _instanceDao.findById(vm.getId());
            if (started != null && started.getState() == VirtualMachine.State.Running) {
+                String message = String.format("HA on VM: %s", started.getHostName());
                s_logger.info("VM is now restarted: " + vmId + " on " + started.getHostId());


Taking the opportunity, what do you think of enhancing also the INFO log message?

As a suggestion, it could be logged something as the following:

HA is now restarting VM instance {id: "123", name: "VM Name", uuid: "1234.....abc", type="User"} on Host [{id: "1", name: "host.name", uuid: "ABC...123", type="Routing"}]

With the following code changes:

Suggested change

s_logger.info("VM is now restarted: " + vmId + " on " + started.getHostId());

HostVO hostVmHasStarted = _hostDao.findById(started.getHostId());

s_logger.info(String.format("HA is now restarting %s on %s", started, hostVmHasStarted));

DaanHoogland · 2022-01-10T10:09:13Z

@blueorangutan package

blueorangutan · 2022-01-10T10:10:03Z

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-01-10T11:45:58Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✖️ suse15. SL-JID 2157

DaanHoogland · 2022-01-10T13:35:37Z

@blueorangutan test

sureshanaparti · 2022-01-10T14:16:01Z

@blueorangutan test

blueorangutan · 2022-01-10T14:19:03Z

@sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-01-10T23:28:38Z

Trillian test result (tid-2845)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31418 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5664-t2845-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

sureshanaparti · 2022-01-11T06:56:50Z

HI @ravening Can you address outstanding comments. Thanks.

nvazquez · 2022-02-06T02:41:35Z

Ping @ravening

When ha is performed on vm's send the alert for it so that its for admins to know which vm's got ha'ed else its time consuming to get those details from logs

ravening · 2022-02-09T13:45:05Z

Ping @ravening

@nvazquez done

nvazquez · 2022-02-09T18:27:30Z

@blueorangutan package

blueorangutan · 2022-02-09T18:28:03Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-02-09T18:57:09Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✔️ suse15. SL-JID 2545

GabrielBrascher

Code LGTM

nvazquez · 2022-02-18T02:20:48Z

@blueorangutan package

blueorangutan · 2022-02-18T02:22:03Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-02-18T03:03:42Z

Packaging result: ✔️ el7 ✖️ el8 ✔️ debian ✔️ suse15. SL-JID 2647

nvazquez · 2022-03-16T03:14:20Z

@blueorangutan package

blueorangutan · 2022-03-16T03:15:03Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-03-16T04:28:47Z

Packaging result: ✖️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2895

nvazquez · 2022-03-16T04:30:18Z

@blueorangutan package

blueorangutan · 2022-03-16T04:31:03Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-03-16T05:08:31Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2898

nvazquez · 2022-03-16T05:15:53Z

@blueorangutan test

blueorangutan · 2022-03-16T05:17:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-03-16T14:46:53Z

Trillian test result (tid-3628)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32812 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5664-t3628-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland approved these changes Nov 4, 2021

View reviewed changes

yadvr approved these changes Nov 18, 2021

View reviewed changes

yadvr approved these changes Dec 30, 2021

View reviewed changes

yadvr added this to the 4.17.0.0 milestone Dec 30, 2021

GabrielBrascher reviewed Dec 30, 2021

View reviewed changes

apache deleted a comment from yadvr Jan 10, 2022

apache deleted a comment from blueorangutan Jan 10, 2022

apache deleted a comment from sureshanaparti Jan 10, 2022

apache deleted a comment from blueorangutan Jan 10, 2022

nvazquez assigned ravening Feb 6, 2022

nvazquez added the type:enhancement label Feb 6, 2022

Rakesh Venkatesh added 2 commits February 9, 2022 14:37

alert: Send alert for ha'ed vm's

165e3b2

When ha is performed on vm's send the alert for it so that its for admins to know which vm's got ha'ed else its time consuming to get those details from logs

feedback change

b905d57

ravening force-pushed the ha_alerts branch from ae80614 to b905d57 Compare February 9, 2022 13:44

GabrielBrascher approved these changes Feb 9, 2022

View reviewed changes

nvazquez approved these changes Mar 17, 2022

View reviewed changes

nvazquez merged commit 6f3c18f into apache:main Mar 17, 2022

	s_logger.info("VM is now restarted: " + vmId + " on " + started.getHostId());
	HostVO hostVmHasStarted = _hostDao.findById(started.getHostId());
	s_logger.info(String.format("HA is now restarting %s on %s", started, hostVmHasStarted));

Conversation

ravening commented Nov 4, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

ravening commented Nov 4, 2021

Uh oh!

DaanHoogland commented Nov 4, 2021

Uh oh!

yadvr commented Nov 18, 2021

Uh oh!

blueorangutan commented Nov 18, 2021

Uh oh!

blueorangutan commented Nov 18, 2021

Uh oh!

yadvr commented Nov 23, 2021

Uh oh!

blueorangutan commented Nov 23, 2021

Uh oh!

blueorangutan commented Nov 23, 2021

Uh oh!

yadvr commented Dec 30, 2021

Uh oh!

blueorangutan commented Dec 30, 2021

Uh oh!

blueorangutan commented Dec 30, 2021

Uh oh!

GabrielBrascher left a comment

Choose a reason for hiding this comment

Uh oh!

GabrielBrascher Dec 30, 2021

Choose a reason for hiding this comment

Uh oh!

sureshanaparti Jan 11, 2022

Choose a reason for hiding this comment

Uh oh!

GabrielBrascher Dec 30, 2021

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Jan 10, 2022

Uh oh!

blueorangutan commented Jan 10, 2022

Uh oh!

blueorangutan commented Jan 10, 2022

Uh oh!

DaanHoogland commented Jan 10, 2022

Uh oh!

sureshanaparti commented Jan 10, 2022

Uh oh!

blueorangutan commented Jan 10, 2022

Uh oh!

blueorangutan commented Jan 10, 2022

Uh oh!

sureshanaparti commented Jan 11, 2022

Uh oh!

nvazquez commented Feb 6, 2022

Uh oh!

ravening commented Feb 9, 2022

Uh oh!

nvazquez commented Feb 9, 2022

Uh oh!

blueorangutan commented Feb 9, 2022

Uh oh!

blueorangutan commented Feb 9, 2022

Uh oh!

GabrielBrascher left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Feb 18, 2022

Uh oh!

blueorangutan commented Feb 18, 2022

Uh oh!

blueorangutan commented Feb 18, 2022

Uh oh!